Method and apparatus for implementing a panoptic camera system

ABSTRACT

A panoptic camera system that can be used to capture all the light from a hemisphere viewing angle is disclosed. The panoptic camera comprises a main reflecting mirror that reflects light from an entire hemisphere onto an image capture mechanism. The main reflecting mirror consists of a paraboloid shape with a dimple on an apex. The surface area around the dimple allows the main reflector to capture light from behind an image capture mechanism or a second reflector. When two panoptic camera systems that capture the light from an entire hemisphere are placed back to back, a camera system that “sees” light from all directions is created. A stereo vision panoramic camera system is also disclosed. The stereo vision panoramic camera system comprises two panoramic camera systems that are separated by a known distance. The two panoramic camera systems are each placed in a “blind spot” of the other panoramic camera system. By using the different images generated by the two panoramic camera systems and the known distance between the two panoramic camera systems, the range to objects within the panoramic images can be determined.

FIELD OF THE INVENTION

The present invention relates to the field of film and videophotography. In particular the present invention discloses a panopticcamera device that captures virtually all the light that converges on asingle point in space.

BACKGROUND OF THE INVENTION

Most cameras only record a small viewing angle. Thus, a typicalconventional camera only captures an image in the direction that thecamera is aimed. Such conventional cameras force viewers to look only atwhat the camera operator chooses to focus on. Some cameras use aspecialized wide angle lens or “fish-eye” lens to capture a widerpanoramic image. However, such panoramic cameras still have a relativelylimited field.

In many situations, it would be much more desirable to have a camerasystem that captures light from all directions. For example, aconventional surveillance camera can be compromised by a perpetratorthat approaches the camera from a direction that is not within theviewing angle of the camera. An ideal surveillance camera would capturelight from all directions such that the camera would be able to recordan image of a person that approaches the camera from any direction.

It would be desirable to have a camera system that would capture thelight from all directions such that a full 360 degree panoramic imagecan be created. A full 360 degree panoramic image would allow the viewerto choose what she would like to look at. Furthermore, a full 360 degreepanoramic image allows multiple viewers to simultaneously view the worldfrom the same point, with each being able to independently choose theirviewing direction and field of view.

SUMMARY OF THE INVENTION

The present invention introduces a panoptic camera system that can beused to capture all the light from a hemisphere viewing angle. Thepanoptic camera comprises a main reflecting mirror that reflects lightfrom an entire hemisphere onto an image capture mechanism. The mainreflecting mirror consists of a paraboloid shape with a dimple on anapex. The surface area around the dimple allows the main reflector tocapture light from behind an image capture mechanism or a secondreflector. When two panoptic camera systems that capture the light froman entire hemisphere are placed back to back, a camera system that“sees” light from all directions is created.

A stereo vision panoramic camera system is also disclosed. The stereovision panoramic camera system comprises two panoramic camera systemsthat are separated by a known distance. The two panoramic camera systemsare each placed in a “blind spot” of the other panoramic camera system.By using the different images generated by the two panoramic camerasystems and the known distance between the two panoramic camera systems,the range to objects within the panoramic images can be determined.

Other objects feature and advantages of present invention will beapparent from the company drawings and from the following detaileddescription that follows below.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will beapparent to one skilled in the art, in view of the following detaileddescription in which:

FIG. 1 illustrates one embodiment of a panoramic camera system.

FIG. 2a illustrates an annular image that is recorded by the panoramiccamera system of FIG. 1.

FIG. 2b illustrates how the annular image of FIG. 2a appears after ithas been unwrapped by polar to rectangular mapping software.

FIG. 3 graphically illustrates the 360 degree band of light that iscaptured by the panoramic camera system of FIG. 1.

FIG. 4a illustrates an embodiment of a panoramic camera system thatcaptures all the light from a hemisphere above and around the panoramiccamera system.

FIG. 4b is a conceptual diagram used to illustrate the shape of thepanoptic camera system in FIG. 4a.

FIG. 5 illustrates an embodiment of a panoramic camera system thatcaptures light from all directions around the panoramic camera system.

FIG. 6 illustrates a first embodiment of a panoramic camera with stereovision.

FIG. 7 illustrates a second embodiment of a panoramic camera with stereovision.

FIG. 8 illustrates an embodiment of a panoramic camera system thatshields unwanted light and limits the amount of light that reaches theimage plane.

FIG. 9 illustrates an embodiment of a panoramic camera that isconstructed using a solid transparent material such that the innercomponents are protected.

FIGS. 10a and 10 b graphically illustrate a method of locating thecenter of a annular panoramic image.

FIGS. 11a and 11 b illustrate flow diagram describing the method oflocating the center of a annular panoramic image.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A method and apparatus for implementing a panoptic camera is disclosed.The following description, for purposes of explanation, specificnomenclature is set forth to provide a thorough understanding of thepresent invention. However, it will be apparent to one skilled in theart that these specific details are not required in order to practicethe present invention. For example, the present invention has beendescribed with reference to Ethernet based computer networks. However,the same techniques can easily be applied to other types of computernetworks.

A Panoptic Camera

FIG. 1 illustrates a cross section view of panoramic camera system 100that captures an image of the surrounding panorama. It should be notedthat the camera system is cylindrically symmetrical such that itcaptures light from a 360 degree band around a point.

The panoramic camera system 100 operates by reflecting all the lightfrom a 360 degree band with a parabolic reflector 110 to a secondreflector 115 through a set of lenses 120, 130, and 140 to an imagecapture mechanism 150. The set of lenses corrects various opticalartifacts created by the parabolic mirror. The image capture mechanism150 may be a chemical based film image capture mechanism or anelectronic based image capture mechanism such as a CCD. Details on howto construct such a panoramic camera can be found in the U.S. patentapplication titled “Panoramic Camera” filed on May 8, 1997, with Ser.No. 08/853,048.

FIG. 2a illustrates how an image captured by the panoramic camera system100 of FIG. 1 appears. As illustrated in FIG. 2a, the surroundingpanorama is captured as an annular image on a two dimensional surface.The annular image can later be processed by an optical or electronicimage processing system to display the image in a more familiar format.FIG. 2b illustrates how the annular image of FIG. 2a appears after ithas been geometrically transformed from the annular image into arectangular image by image processing software. In one embodiment, thetransformation approximates a transform from polar coordinates torectangular coordinate.

FIG. 3 graphically illustrates the band of light that is captured by thepanoramic camera system of FIG. 1. As illustrated in FIG. 3, thepanoramic camera system of FIG. 1 captures a 360 degree band of lightthat is 60 degrees above and below the horizon.

Camera System that Collects all Light from a Hemisphere

In certain applications, it would be desirable to have a camera systemthat collects all the light from a full hemisphere around the camera.For example, a camera system that collects all the light from a fullhemisphere could be used by astronomers to capture an image of theentire night sky.

FIG. 4a illustrates a camera system similar to camera system of FIG. 1except that the camera system of FIG. 4a captures light from the horizonline all the way to the Zenith. Thus, the camera system of FIG. 4acaptures light from the entire hemisphere above the camera system.

The camera system operates by having a main reflector 410 that reflectslight from the entire hemisphere above the camera system to a secondreflector 415. The second reflector 415 reflects the light down througha lens system 420 to an image capture mechanism 440.

To be able to collect light from a full hemisphere, the main reflectorof the camera system consists a cylindrically symmetric mirror with across section that consists of an offset parabola. FIG. 4b illustratesthe shape of a full parabola 450 that is then cut shortly after the apexon the side of the parabola near the center of the main reflector. Theoffset parabola reflects light from a slightly greater than 90 degreeband that starts at the horizon (see light ray 481) and continues to thezenith (see light rays 485 and 489) and beyond. The short section ofparabola near the center of the main reflector allows the main reflectorto direct light from the zenith and beyond to the second reflector 470and down into the image capture mechanism 440. This is illustrated bylight ray 489.

Although the main reflector 410 of FIG. 4a captures light from thezenith and beyond, the main reflector has a slight “blind spot.” Theblind spot is limited to being a small cone of space behind the secondreflector 415 inside light ray 439. This small area in the blind spotcan be used to implement a support fixture for the mirror.Alternatively, the small area in the blind spot can be used to implementsupplemental lighting.

Camera System that Collects Light from all Directions

For some applications, it would be desirable to have a camera systemthat collects all the light that converges on a point from alldirections. For example, an ideal security camera system would be ableto “see” in all directions such that no perpetrator could sneak up onthe camera from an angle not seen by the camera. Thus, no perpetratorcould sneak up on the camera and disable the camera without having hisimage captured by the camera.

To construct a panoptic camera system that collects light from alldirections, the present invention discloses an arrangement of twohemisphere camera systems joined together as illustrated in FIG. 5. Thearrangement of FIG. 5 will produce two annular images: one annular imagefor the upper hemisphere and one annular image for the lower hemisphere.Since the two camera systems are aligned with each other, the twoannular images can be optically or electronically combined to generatean image of the entire surroundings of the panoptic camera system.

A Panoramic Camera System with Stereo Vision

To gauge the range of visible objects, humans uses stereo vision.Specifically, the two different view angles provided by two eyes enablesa human to determine the relative distance of visible objects. The sameprinciple can be used to implement a panoramic camera system that hasstereo vision.

A First Embodiment

Referring back to FIG. 1, the original panoramic camera has a blind spotabove the second reflector. The blind spot is clearly illustrated inFIG. 3 wherein the area above 60 degrees above the horizon and the areabelow 60 degrees below the horizon are not captured by the panoramiccamera system. A second panoramic camera can be placed in a blind spotof a first panoramic camera. FIG. 6 illustrates a stereo visionpanoramic camera system constructed according to this technique.

The stereo panoramic camera system of FIG. 6 comprises a first panoramiccamera 605 and a second inverted panoramic camera 635. Each panoramiccamera system 605 and 635 is in a blind spot of the other panoramiccamera system. By spatially separating the two panoramic camera systems,each panoramic camera system will record a slightly different annularimage of the surrounding panorama. Using the known distance between thetwo panoramic camera systems and the two different annular images, thedistance to objects within the annular images can be determined.

In the embodiment displayed in FIG. 6, the two panoramic camera systemsuse a single dual sided reflector 615 to reflect the panoramic imagefrom the main reflector into the respective image capture mechanisms. Inan alternate embodiment (not shown), two panoramic camera systems can beplaced in the other blind spot such that the two panoramic camerasystems are arranged in a manner similar to the arrangement of FIG. 5.

Another Stereo Vision Embodiment

FIG. 7 illustrates yet another embodiment of a stereo vision panoramiccamera system. In the embodiment of FIG. 7, a single image capturemechanism 750 is used to capture two slightly different panoramicimages.

The stereo vision panoramic camera of FIG. 7 captures a first panoramicannular image using a first main reflector 735 and a second reflector715 in the same manner described with reference to FIG. 1. However, thesecond reflector 715 in the stereo vision panoramic camera system ofFIG. 7 is an electrically activated mirror. The stereo vision panoramiccamera system of FIG. 7 also features a second main reflector 760 thatis positioned at the correct position for an optical path that does notrequire a second reflector. Thus, by deactivating the electricallyactivated second reflector 715, the stereo vision panoramic camerasystem captures a second panoramic annular image using the second mainreflector 760. The two panoramic annular images can be combined todeliver a stereo image of the surrounding panorama.

Panoramic Camera System with Protective Shield

When collecting light reflected off the main reflector of a panoramiccamera system, it is desirable to eliminate any influence from lightfrom other sources. For example, ambient light should not be able toenter the optical system that is intended only to collect the panoramicimage reflected off of the main reflector.

FIG. 8 illustrates a cross-section view of an improved panoramic camerasystem 800 the collects only the light from the reflected panoramicimage. It should be noted that the real panoramic camera system iscylindrically symmetrical. The panoramic camera panoramic camera system800 uses two light shields (875 and 877) to block all light that is notfrom the reflected image off of the main reflector.

The first light shield 875 is mounted on top of the second reflector 815that reflects the panoramic image on the main reflector 810 down intothe optical path of the camera system. The first light shield 875prevents light from above the panoramic camera's maximum verticalviewing angle. In one embodiment, the panoramic camera's maximumvertical viewing angle is 50 degrees such that the first light shield875 prevents light coming from an angle greater than 50 degree fromentering the panoramic camera's optical path.

The second light shield 877 is placed around the opening of thepanoramic camera's lens system. The second light shield prevents lightfrom entering the camera's optical path unless that light has reflectedoff the main reflector 810 and has reflected off the second reflector815 down into the optical path.

FIG. 8 also illustrates that the second reflector 815 can be constructedusing a convex mirror instead of the flat mirror. By using a convexmirror as the second reflector, the second reflector can be placedcloser to the main body of the camera system.

Overexposure Control

A panoramic camera system must be able to handle a much wider variety oflighting conditions than a conventional (limited viewing angle) camerasystem. A conventional camera system only captures light from a smallviewing angle such that the intensity of light from the viewing anglewill probably not vary a great amount. However, a panoramic camerasystem captures light from all directions such that the wide variety oflighting conditions must be handled. For example, with a panoramiccamera system light from a first direction may come directly from thesun and in another light from a second direction may consist of ambientlight reflected off of an object in a shadow. To capture a high qualitypanoramic image, it would be desirable to adjust the amount of lightcaptured from each viewing direction such that the light exposure fromthe different directions does not vary wildly.

FIG. 8 illustrates a panoramic camera constructed to limit the lightreceived from the different directions. To adjust the amount of lightcaptured from each direction, the panoramic camera system 800 includesan adaptive light filter 890 in the optical path of the panoramic camerasystem. The adaptive light filter 890 limits the amount of light thatreaches the image capture mechanism 850.

In the illustration of FIG. 8, the adaptive light filter 890 is placedjust before the image capture mechanism 850. This position minimizes thedetrimental effects caused by any scattering of light by the adaptivelight filter 890. However, the adaptive light filter 890 can be placedat any point in the optical path of the panoramic camera system.

A Passive Filtering System

One method of implementing an adaptive light filter 890 is to use anormally transparent light sensitive material that darkens when thematerial is exposed to large quantities of light. For example, arefractive neutral lens made of photogray material would automaticallylimit the amount of light from high intensity viewing directions.Examples of photogray glass include PhotoGray Extra and PhotoGray IImade by Corning Glass Works of Corning, New York.

An Active Filtering System

Another method of implementing an adaptive light filter 890 is to use anelectronically controlled Liquid Crystal Display (LCD) array as anadaptive light filter 890. Ideally, the LCD array would be capable ofselectively adjusting the amount of light that passes through any pointof the LCD array.

To control the LCD array, an LCD control circuit (not shown) would becoupled to the electronic image capture mechanism 850 of the panoramiccamera system 800. The electronic image capture mechanism 850 woulddetermine the relative light intensity at each point on the electronicimage capture mechanism. The light intensity information from theelectronic image capture mechanism 850 is passed to the LCD controlcircuit that determines how the LCD array should limit the light thatpasses through. Specifically, when the electronic image capturemechanism 850 detects an area that is receiving high intensity light,then the LCD control circuit would darken the corresponding area on theLCD array. Thus, the LCD array would selectively reduce the amount oflight that reaches the image capture mechanism from high light intensitydirections. The “flattening” of the light intensity results in capturedpanoramic annular images with greater contrast.

A Solid Camera Embodiment

The panoramic camera system illustrated in FIG. 8 uses an outer surfacemirror for the main reflector. An outer surface mirror is used since aninner surface mirror protected by a transparent material would haverefractive effects caused when the light enters the transparent materialand when the light exits the transparent material. Since the panoramiccamera system illustrated in FIG. 8 uses an outer surface mirror, thecamera must be used cautiously to prevent damage to the out mirrorsurface. It would therefore be desirable to implement a panoramic camerathat protects the main reflector.

FIG. 9 illustrates an embodiment of a panoramic camera systemconstructed of a solid transparent block. In the embodiment of FIG. 9,the main reflector 910 is protected by a transparent material 912. Thetransparent material 912 is shaped such that all the light that will beused to create the annular reflection of the surrounding panorama enterthe transparent material 912 at a normal to the surface of thetransparent material 912 as illustrated by the right angle marks on thelight rays. Since the light rays that create the annular image enter ata normal to the surface, there is no refractive effect as the lightenters the transparent material 912. The outer surface of thetransparent material 912 is coated with a multicoat material such thatinternal reflections are prevented.

Once a light ray that will form part of the panoramic image enters thetransparent material 912, the light ray then reflects off the mainreflector 910 and then reflects off the second reflector 915 and thenexits the transparent material 912 at surface 920. Thus, the lightremains in the transparent material 912 until it enters the lens system.The surface 920 can be shaped such that all light that is part of theannular image exits at a normal to the surface 920 such that thetransparent material 912 has no refractive effect on the light.Alternatively, the surface 920 can be shaped such that surface 920 ispart of the lens system.

The embodiment in FIG. 9 includes two light shields 975 and 977 toprevent undesired light from entering the optical path. It should benoted that the panoramic camera system can also be constructed with thelight shields 975 and 977.

Annular Image Processing

As previously described, the panoramic annular images can begeometrically transformed from the annular image into more conventionalrectangular projections. One method of performing this operation is touse digital image processing techniques as described in the relate U.S.patent titled “Panoramic Camera” filed on May 8, 1997, with Ser. No.08/853,048.

When photographic film is used to capture the annular images, theannular images will not always be recorded in the exact same position onthe film. One reason for this is that sprockets used to advance filmthrough a camera are slightly smaller that the correspond holes in thefilm. Thus, the film alignment between exposures tends to vary. Thiseffect is known as “gate weave.”

To process an annular image, the center coordinate of the digitizedannular image must be known in order to rotate a selected viewport intoa standard view. Since gate weave causes the center coordinate to vary,the center coordinate must be determined for each annular image thatoriginated from photographic film. FIGS. 10b, 10 b, 11 a and 11 billustrate a method of determining the center coordinate of an digitizedpanoramic annular image that originated from photographic film.

Referring to the flow diagram of FIG. 11a, step 1110 selects an initialproposed center point along a first axis. Referring to FIG. 10a, aninitial proposed center point PC₁ is illustrated along a Y axis (thefirst axis). Next at step 1120, the annular video to standard videoconversion software finds a first pixel along an orthogonal second axisthat passes through the first proposed center point and exceeds athreshold value. In FIG. 10a this is illustrated as FP₁ on a X axis. Asillustrated in FIG. 10a, the threshold value is selected to locate thefirst pixel along the edge of the annular image. Next, a last pixel thatexceeds the threshold and is located along the second axis that passesthrough the first proposed center point (PC₁) is selected. In FIG. 10a,that last pixel is LP₁ along an X axis. Next at step 1130, the converterselects the midpoint between the first pixel FP₁ and the last pixel LP₁along the second axis as a second proposed center point. In FIG. 10a,the second proposed center point is illustrated as PC₂. The secondproposed center point is closer to the actual center than the firstproposed center point.

This process is repeated again after switching axis. Specifically, instep 1140 a first pixel a first axis that passes through the secondproposed center point and exceeds a threshold value is selected as afirst pixel. This is illustrated in FIG. 10b as point FP₁ along a Yaxis. Then a last pixel along a first axis that passes through a secondproposed center point and exceeds the threshold value is selected. InFIG. 10b this is illustrated as LP₂. Then a midpoint is selected betweenthe first pixel FP₂ and the last pixel LP₂ as the third proposed centerpoint. This is illustrated on FIG. 10b as third proposed center pointPC₃. The third proposed center point is also referred to as the firstproposed center point for purposes of repeating the method steps.

The method proceeds to step 1160 where it determines if the first/thirdproposed center point is equal to the second proposed center point. Thistest determines whether the same center point has been selected again.If this occurs, then the method proceeds down to step 1180 where thesecond proposed center point is selected as the center point of theannular image. If the first proposed center point is not the same as thesecond proposed center point the method proceeds to step 1170 where themethod determines if a minimum number of iterations have been performed.If this has not occurred, then the method proceeds back up to 1120 whereit can repeat additional iterations of the method to determine a moreaccurate center point.

The foregoing disclosure has described several panoramic cameraembodiments. It is contemplated that changes and modifications may bemade by one of ordinary skill in the art, to the materials andarrangements of elements of the present invention without departing fromthe scope of the invention.

We claim:
 1. A camera apparatus, said camera apparatus comprising: animage capture mechanism; and a main reflector, said main reflectorreflecting light from a full hemisphere view onto said image capturemechanism; wherein said main reflector comprises a cylindricallysymmetrical shape of a parabola segment rotated about an axis, saidparabola segment comprising a vertex, a first side of said parabolasegmenty, and a second side of said parabola segment shorter than saidfirst side and adjacent to said axis.
 2. The apparatus as claimed inclaim 1 further comprising: a second reflector, said second reflectorpositioned such that said light is reflected from said main reflectoronto said second reflector and then from said second reflector onto saidimage capture mechanism.
 3. The apparatus as claimed in claim 1 whereinsaid light passes through a set of lenses before landing on said imagecapture mechanism.
 4. A camera apparatus, said camera apparatuscomprising: an image capture mechanism; and a main reflector, said mainreflector comprising a paraboloid shape with a dimple on an apex;wherein said main reflector comprises a cylindrically symmetrical shapeof a parabola segment rotated about an axis, said parabola segmentcomprising a vertex, a first side of said parabola segment, and a secondside of said parabola segment shorter than said first side and adjacentto said axis.
 5. The apparatus as claimed in claim 4 further comprising:a second reflector, said second reflector positioned such that saidlight is reflected from said main reflector onto said second reflectorand then from said second reflector onto said image capture mechanism.6. The apparatus as claimed in claim 5 wherein said light passes througha set of lenses before landing on said image capture mechanism.